Block Coordinate Descent for Deep Learning: Unified Convergence Guarantees

Training deep neural networks (DNNs) efficiently is a challenge due to the associated highly nonconvex optimization. Recently, the efficiency of the block coordinate descent (BCD) type methods has been empirically illustrated for DNN training. The main idea of BCD is to decompose the highly composite and nonconvex DNN training problem into several almost separable simple subproblems. However, their convergence property has not been thoroughly studied. In this paper, we establish some unified global convergence guarantees of BCD type methods for a wide range of DNN training models, including but not limited to multilayer perceptrons (MLPs), convolutional neural networks (CNNs) and residual networks (ResNets). This paper nontrivially extends the existing convergence results of nonconvex BCD from the smooth case to the nonsmooth case. Our convergence analysis is built upon the powerful Kurdyka-Łojasiewicz (KL) framework but some new techniques are introduced, including the establishment of the KL property of the objective functions of many commonly used DNNs, where the loss function can be taken as squared, hinge and logistic losses, and the activation function can be taken as rectified linear units (ReLUs), sigmoid and linear link functions. The efficiency of BCD method is also demonstrated by a series of exploratory numerical experiments.